Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add sanity test script #878

Merged
merged 9 commits into from
Nov 11, 2024
Merged

Add sanity test script #878

merged 9 commits into from
Nov 11, 2024

Conversation

qianheng-aws
Copy link
Contributor

@qianheng-aws qianheng-aws commented Nov 7, 2024 •
edited
Loading

Description

This Python script executes test queries from a CSV file using an asynchronous query API and generates comprehensive test reports.

The script produces two report types:

  1. An Excel report with detailed test information for each query
  2. A JSON report containing both test result overview and query-specific details

Apart from the basic feature, it also has some advanced functionality includes:

  1. Concurrent query execution (note: the async query service has session limits, so use thread workers moderately despite it already supports session ID reuse)
  2. Configurable query timeout with periodic status checks and automatic cancellation if timeout occurs.
  3. Flexible row selection from the input CSV file, by specifying start row and end row of the input CSV file.
  4. Expected status validation when expected_status is present in the CSV
  5. Ability to generate partial reports if testing is interrupted
    An example to run the test script:
python SanityTest.py --base-url ${URL_ADDRESS} --username *** --password *** --datasource ${DATASOURCE_NAME} --input-csv test_queries.csv --output-file test_report --max-workers 2 --check-interval 10 --timeout 600

For more details, you can see the help manual via command:

python SanityTest.py --help   

usage: SanityTest.py [-h] --base-url BASE_URL --username USERNAME --password PASSWORD --datasource DATASOURCE --input-csv INPUT_CSV
                                      --output-file OUTPUT_FILE [--max-workers MAX_WORKERS] [--check-interval CHECK_INTERVAL] [--timeout TIMEOUT]
                                      [--start-row START_ROW] [--end-row END_ROW]

Run tests from a CSV file and generate a report.

options:
  -h, --help            show this help message and exit
  --base-url BASE_URL   Base URL of the service
  --username USERNAME   Username for authentication
  --password PASSWORD   Password for authentication
  --datasource DATASOURCE
                        Datasource name
  --input-csv INPUT_CSV
                        Path to the CSV file containing test queries
  --output-file OUTPUT_FILE
                        Path to the output report file
  --max-workers MAX_WORKERS
                        optional, Maximum number of worker threads (default: 2)
  --check-interval CHECK_INTERVAL
                        optional, Check interval in seconds (default: 10)
  --timeout TIMEOUT     optional, Timeout in seconds (default: 600)
  --start-row START_ROW
                        optionl, The start row of the query to run, start from 0
  --end-row END_ROW     optional, The end row of the query to run

Related Issues

Resolves #877

Check List

  • Updated documentation (docs/ppl-lang/README.md)
  • Implemented unit tests
  • Implemented tests for combination with other commands
  • New added source code should include a copyright header
  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@qianheng-aws
Add sanity test script
550b516 
Signed-off-by: Heng Qian <[email protected]>
@qianheng-aws
Add header
86ba9c8 
Signed-off-by: Heng Qian <[email protected]>
@qianheng-aws
Minor fix
46b72b4 
Signed-off-by: Heng Qian <[email protected]>
@qianheng-aws
Minor fix
4d673e3 
Signed-off-by: Heng Qian <[email protected]>
@qianheng-aws
Minor fix
08ebf9a 
Signed-off-by: Heng Qian <[email protected]>
@YANG-DB
Copy link
Member

YANG-DB commented Nov 7, 2024 •
edited
Loading

@qianheng-aws this looks good - please add

  • documentation with all the above description and instructions + results report example
  • what are the prerequisites for running the script - docker ?
  • what are the data prerequisites for running the script - import content process ?
  • export the results to a file including :
    • ppl version
    • performance time
    • coverage
  • plz also add a folder for CSV tests

@qianheng-aws
Copy link
Contributor Author

qianheng-aws commented Nov 11, 2024 •
edited
Loading

@qianheng-aws this looks good - please add

  • documentation with all the above description and instructions + results report example

  • what are the prerequisites for running the script - docker ?

  • what are the data prerequisites for running the script - import content process ?

  • export the results to a file including :

    • ppl version
    • performance time
    • coverage

  • plz also add a folder for CSV tests

I addressed most part of the comment and add them in the README.md, except some points:

  • the data prerequisites. I haven't got the background of where the data comes from and how to ingest them to flint yet. Left a TODO in the README.
  • ppl version. Do you mean the version returns from API GET _cat/plugins?v? I think it should always be the same as the opensearch version. Will it help to add that in the report?

@qianheng-aws
Minor fix
e23aee7 
Signed-off-by: Heng Qian <[email protected]>
@qianheng-aws
Support set log-level
7afc012 
Signed-off-by: Heng Qian <[email protected]>
YANG-DB
YANG-DB approved these changes Nov 11, 2024
View reviewed changes
Copy link
Member

@YANG-DB YANG-DB left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice 👍

@YANG-DB YANG-DB merged commit a0c246b into opensearch-project:main Nov 11, 2024
4 checks passed
@qianheng-aws qianheng-aws deleted the sanityTest branch November 12, 2024 05:31
kenrickyap pushed a commit to Bit-Quill/opensearch-spark that referenced this pull request Dec 11, 2024
@qianheng-aws @14yapkc1
Add sanity test script (opensearch-project#878)
e475b77 
* Add sanity test script

Signed-off-by: Heng Qian <[email protected]>

* Add header

Signed-off-by: Heng Qian <[email protected]>

* Minor fix

Signed-off-by: Heng Qian <[email protected]>

* Minor fix

Signed-off-by: Heng Qian <[email protected]>

* Minor fix

Signed-off-by: Heng Qian <[email protected]>

* Support check expected_status if have that column in input file.

Signed-off-by: Heng Qian <[email protected]>

* Add README.md

Signed-off-by: Heng Qian <[email protected]>

* Minor fix

Signed-off-by: Heng Qian <[email protected]>

* Support set log-level

Signed-off-by: Heng Qian <[email protected]>

---------

Signed-off-by: Heng Qian <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@YANG-DB YANG-DB YANG-DB approved these changes

@LantaoJin LantaoJin LantaoJin approved these changes

@dai-chen dai-chen Awaiting requested review from dai-chen dai-chen is a code owner

@mengweieric mengweieric Awaiting requested review from mengweieric mengweieric is a code owner

@vmmusings vmmusings Awaiting requested review from vmmusings

@penghuo penghuo Awaiting requested review from penghuo penghuo is a code owner

@seankao-az seankao-az Awaiting requested review from seankao-az seankao-az is a code owner

@anirudha anirudha Awaiting requested review from anirudha anirudha is a code owner

@kaituo kaituo Awaiting requested review from kaituo kaituo is a code owner

@noCharger noCharger Awaiting requested review from noCharger noCharger is a code owner

@ykmr1224 ykmr1224 Awaiting requested review from ykmr1224 ykmr1224 is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[FEATURE] Support running sanity test automatically
3 participants
@qianheng-aws @YANG-DB @LantaoJin